469 research outputs found
Learning recurrent representations for hierarchical behavior modeling
We propose a framework for detecting action patterns from motion sequences
and modeling the sensory-motor relationship of animals, using a generative
recurrent neural network. The network has a discriminative part (classifying
actions) and a generative part (predicting motion), whose recurrent cells are
laterally connected, allowing higher levels of the network to represent high
level phenomena. We test our framework on two types of data, fruit fly behavior
and online handwriting. Our results show that 1) taking advantage of unlabeled
sequences, by predicting future motion, significantly improves action detection
performance when training labels are scarce, 2) the network learns to represent
high level phenomena such as writer identity and fly gender, without
supervision, and 3) simulated motion trajectories, generated by treating motion
prediction as input to the network, look realistic and may be used to
qualitatively evaluate whether the model has learnt generative control rules
Multi-camera Realtime 3D Tracking of Multiple Flying Animals
Automated tracking of animal movement allows analyses that would not
otherwise be possible by providing great quantities of data. The additional
capability of tracking in realtime - with minimal latency - opens up the
experimental possibility of manipulating sensory feedback, thus allowing
detailed explorations of the neural basis for control of behavior. Here we
describe a new system capable of tracking the position and body orientation of
animals such as flies and birds. The system operates with less than 40 msec
latency and can track multiple animals simultaneously. To achieve these
results, a multi target tracking algorithm was developed based on the Extended
Kalman Filter and the Nearest Neighbor Standard Filter data association
algorithm. In one implementation, an eleven camera system is capable of
tracking three flies simultaneously at 60 frames per second using a gigabit
network of nine standard Intel Pentium 4 and Core 2 Duo computers. This
manuscript presents the rationale and details of the algorithms employed and
shows three implementations of the system. An experiment was performed using
the tracking system to measure the effect of visual contrast on the flight
speed of Drosophila melanogaster. At low contrasts, speed is more variable and
faster on average than at high contrasts. Thus, the system is already a useful
tool to study the neurobiology and behavior of freely flying animals. If
combined with other techniques, such as `virtual reality'-type computer
graphics or genetic manipulation, the tracking system would offer a powerful
new way to investigate the biology of flying animals.Comment: pdfTeX using libpoppler 3.141592-1.40.3-2.2 (Web2C 7.5.6), 18 pages
with 9 figure
Detecting the Starting Frame of Actions in Video
In this work, we address the problem of precisely localizing key frames of an
action, for example, the precise time that a pitcher releases a baseball, or
the precise time that a crowd begins to applaud. Key frame localization is a
largely overlooked and important action-recognition problem, for example in the
field of neuroscience, in which we would like to understand the neural activity
that produces the start of a bout of an action. To address this problem, we
introduce a novel structured loss function that properly weights the types of
errors that matter in such applications: it more heavily penalizes extra and
missed action start detections over small misalignments. Our structured loss is
based on the best matching between predicted and labeled action starts. We
train recurrent neural networks (RNNs) to minimize differentiable
approximations of this loss. To evaluate these methods, we introduce the Mouse
Reach Dataset, a large, annotated video dataset of mice performing a sequence
of actions. The dataset was collected and labeled by experts for the purpose of
neuroscience research. On this dataset, we demonstrate that our method
outperforms related approaches and baseline methods using an unstructured loss
- …